feat: add URL Inspector PG API endpoints | LLMO-4030#2113
feat: add URL Inspector PG API endpoints | LLMO-4030#2113
Conversation
4 new endpoints calling the URL Inspector RPCs in mysticat-data-service: - GET .../url-inspector/stats — aggregate stats + weekly sparklines - GET .../url-inspector/owned-urls — paginated owned URL citations - GET .../url-inspector/trending-urls — paginated non-owned URLs - GET .../url-inspector/cited-domains — domain-level aggregations Exports shared utilities from llmo-brand-presence.js for reuse. LLMO-4030 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
This PR will trigger a minor release when merged. |
…e-retrieve-LLMO-4030 Made-with: Cursor # Conflicts: # src/controllers/llmo/llmo-brand-presence.js # src/controllers/llmo/llmo-mysticat-controller.js
Made-with: Cursor
…O-4030 Add urlId field to domain-urls handler response mapping so the UI can pass it to the url-prompts endpoint for Phase 3 drilldown. Made-with: Cursor
… LLMO-4030 Replace broken parseBody(response) helper with standard response.json() to match the Web Response API used by spacecat-shared-http-utils. Add comprehensive tests for all error paths (model validation, RPC errors, missing params, site-org validation) and null-field handling to reach 100% line/branch/statement/function coverage. Made-with: Cursor
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…- LLMO-4030 Map the new prompts_cited, categories, and regions fields from the enriched rpc_url_inspector_domain_urls RPC into the API response. Made-with: Cursor
Code ReviewNice work on this PR — the architecture is clean, the plan doc is excellent, and the test coverage is thorough. A few items to flag: Issues1. Stale JSDoc on The comment says "No pagination — domain count per site is bounded" but the handler actually implements pagination via 2. The 3. The domain-urls RPC call omits category and region filters that the parent endpoints (cited-domains, stats) do pass. If a user has filtered by category/region at the domain level and then drills into a domain, those filters would silently be dropped. Same applies to 4. RPC errors exposed to clients All handlers return Minor
|
- Fix stale JSDoc on cited-domains handler (now paginated) - Document why domain-urls and url-prompts don't pass brandId/category/region (underlying RPCs don't accept these params; filtering is at parent level) - Return internalServerError for RPC failures instead of leaking raw PostgreSQL error messages to clients (details remain in server logs) - Add total_count to cited-domains test happy path data - Filter out null URL rows in trending-urls grouping Made-with: Cursor
Summary-table RPCs no longer accept p_brand_id (brand_id is not in the summary table). Update handlers and tests accordingly. Made-with: Cursor
Add test for trending URL rows with valid url but null content_type, prompt, category, region, topics, citation_count, execution_count, and total_non_owned_urls to reach 100% branch coverage. Made-with: Cursor
Document the 6 URL Inspector API endpoints under the org-scoped brand-presence path: - stats: aggregate citation stats + weekly sparklines - owned-urls: paginated owned URL citations with WoW trends - trending-urls: paginated non-owned URLs grouped by URL - cited-domains: domain-level citation aggregations - domain-urls: Phase 2 drilldown into domain URLs - url-prompts: Phase 3 drilldown into prompts per URL Adds response schemas and path references. Validated with docs:lint and docs:build. Made-with: Cursor
…e-retrieve-LLMO-4030 Resolve merge conflict in docs/index.html by regenerating from merged OpenAPI sources. Made-with: Cursor
Addresses review feedback asking for brandalf-style .md docs for the new URL Inspector endpoints so AI / on-call engineers have a reliable reference. Adds one doc per endpoint (stats, owned-urls, trending-urls, cited-domains, domain-urls, url-prompts) plus a consolidated url-inspector-apis-overview.md, and links the new overview from brand-presence-apis-overview.md. Each doc covers path shape, query params (incl. aliases and defaults), RPC signature with conceptual SQL, response shape, sample URLs, error responses, and auth/access expectations. Made-with: Cursor
…-4030 Replace the single rpc_url_inspector_stats call with a Promise.all fanout across the four per-KPI RPCs that land in mysticat-data-service (total_prompts, total_prompts_cited, unique_urls, total_citations). The response shape is unchanged; the controller unions weeks across the four streams and keeps missing metrics at 0. Thread ctx.params.brandId through as p_brand_id on every call (with 'all' -> NULL), matching the pattern the other URL Inspector handlers already use. This enables proper brand scoping, which the old summary-table RPC could not support. End-to-end latency is now max-of-four across the RPCs instead of sum-of- four (or the 60s+ of the broken summary RPC), so ~1.5s warm on adobe.com 28-day without a brand filter, ~3.5s with one (bounded by total_prompts). Also: - Wrap Promise.all in try/catch so thrown exceptions log and return 500 cleanly instead of crashing the handler. - Enrich the RPC-error log with PostgREST code/details/hint so the next failure is easier to diagnose. - Rewrite the stats tests to target the four split RPCs, add a statsRpcResults() helper, and add coverage for brandId threading, parallel fanout, and the union-of-weeks reassembly. - Rewrite the URL Inspector Stats API doc to describe the fanout, the p_brand_id semantics, and the plan-shape rule. See mysticat-data-service/docs/plans/2026-04-02-url-inspector-performance.md Experiment 6 for the rationale and benchmarks. Made-with: Cursor
…dler - LLMO-4030 The stats handler's try/catch around Promise.all and the code/details/hint string-building in the RPC-error log were uncovered, dropping coverage below the 100% global threshold. Add four focused tests: one for message-only PostgREST errors (exercises the falsy branches of the code/details/hint ternaries), one with all three populated (truthy branches), one for a rejected Promise with an Error instance, and one for a bare-string rejection (exercises the `e?.message || e` fallback). Made-with: Cursor
Resolve docs/index.html conflict by regenerating via npm run docs:build against the merged docs/openapi/ sources (auto-merged cleanly). All other files auto-merged without conflicts. Made-with: Cursor
The pinned v1.56.0 tag has been evicted from ECR (ECR lifecycle retention), causing ci/it-postgres to fail on both main and this branch with "manifest unknown". Bump to v1.67.8 (released 2026-04-21, latest available image) to restore integration tests. This is a fix for a pre-existing main-branch issue that surfaced on this PR after the last merge; landing it here unblocks both branches. LLMO-4030 Made-with: Cursor
Summary
GET .../url-inspector/stats— aggregate stats + weekly sparklinesGET .../url-inspector/owned-urls— paginated owned URL citationsGET .../url-inspector/trending-urls— paginated non-owned URLs (groups flat RPC rows by URL)GET .../url-inspector/cited-domains— domain-level aggregationsGET .../url-inspector/domain-urls— paginated URLs within a domain (with urlId, promptsCited, categories, regions)GET .../url-inspector/url-prompts— prompt breakdown for a specific URLllmo-brand-presence.jsfor reuseKey changes since initial PR
domain-urlsresponse now includesurlId,promptsCited,categories, andregionsfields from the enriched RPCurl-promptsendpoint added for Phase 3 drilldown (prompt analysis per URL)Related PRs
Test plan